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ABSTRACT 

The  paper  deals  with  a  system  of  linear  inequalities  in  a  finite  dimension 
al  space.  When  the  system  is  inconsistent,  we  are  interested  in  vectors  that 
satisfy  the  system  in  a  least-squares  sense.  We  characterize  such  least- 
squares  solutions  and  propose  a  method  to  find  one  of  these  solutions.  It  is 
shown  that  for  any  given  system  of  linear  inequalities  and  for  any  starting 
point,  the  method  can  produce  a  solution  in  a  finite  number  of  iterations. 
Computational  results  are  very  satisfactory  in  terms  of  accuracy  and  number 
of  iterations. 
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SIGNIFICANCE  AND  EXPLANATION 


The  linear  least-square  problem  for  equations  has  been  thoroughly 
studied  and  efficient  computational  methods  for  solving  it  exist.  In  this 
paper  we  consider  its  extension  to  the  case  of  linear  inequalities.  Such  a 
problem  formulation  can  be  used  to  find  a  least-squares  solution  to  a  linear 
programming  problem,  even  when  the  linear  programming  problem  has  no  solution. 

We  characterize  such  least-squares  solutions  and  propose  a  method  to 
compute  one  such  solution.  It  is  shown  that  for  any  given  system  of 
linear  inequalities  the  method  can  produce  a  least-squares  solution  in  a 
finite  number  of  steps.  Numerical  results  for  the  method  are  extremely 
satisfactory. 


The  responsibility  for  the  wording  and  views  expressed  in  this  descriptive 
summary  lies  with  MRC,  and  not  with  the  author  of  this  report. 


LEAST-SQUARES  SOLUTION  OF  LINEAR  INEQUALITIES 


S.-P.  Han 

1.  INTRODUCTION 

We  are  concerned  with  a  system  of  linear  inequalities  Ax  <  b,  where  A 
is  an  m  x  n  matrix  and  b  an  m-vector.  When  the  system  Ax  <  b  is  in¬ 
consistent,  as  in  the  case  of  equalities,  it  is  often  desirable  to  have  a 
vector  that  satisfies  the  system  in  the  least-squares  sense.  More  specifically 
we  are  interested  in  a  vector  x  that  minimizes  the  quantity  #  (Ax  -  bj^i^, 
where  (Ax  -  b)+  is  the  m-vector  whose  i  -th  component  is  max  {(Ax  -  b)^,  0} 

The  problem  to  find  a  least-squares  solution  to  the  system  Ax  <  b  is 
fundamental.  It  is  a  natural  extension  of  the  equality  linear  least-squares 
problem  and  abounds  with  applications.  For  instance,  a  least-squares  solution 
to  the  system  derived  from  the  Karush-Kuhn-Tucker  conditions  of  a  linear 
program  can  often  supply  useful  information  even  when  the  given  linear 
program  is  infeasible.  A  technique  for  solving  such  a  least-squares  problem 
can  also  be  used  to  check  the  feasibility  or  to  find  a  feasible  point  for  a 
linearly  constrained  problem. 

There  are  basically  two  direct  approaches  for  tackling  this  problem. 

The  first  one  is  to  use  an  unconstrained  minimization  method  to  minimize  the 
following  function 

f  (x)  :=  £-(Ax  -  b)^(Ax  -  b)+  (1.1) 


Unfortunately,  the  function  f  is  not  twice  differentiable  and  the  powerful 
Newton-like  methods  are  not  applicable. 

The  second  approach  is  to  transform  the  problem  into  its  equivalent 


quadratic  programming  problem 


min 

(x,z) 


s . t .  Ax  -  b  <  z . 


(1.2) 
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But,  most  general  quadratic  programming  techniques  are  not  very  suitable  for 
the  problem  (1.2)  because  they  fail  to  exploit  its  special  structure.  Further¬ 
more,  since  the  solutions  to  (1.2)  are  usually  not  unique,  some  numerical 
difficulties  may  also  arise. 

For  the  least-squares  problem  we  propose  in  this  paper  a  method  that 
can  avoid  the  difficulties  mentioned  above.  The  method  is  iterative  in  nature 
but  has  a  finite  convergence  property.  Its  numerical  results  are  extremely 
good. 

We  note  here  that  the  theorems  and  the  method  considered  in  this  paper 
can  be  straightforwardly  extended  to  the  mixed-type  system  Ax  <  b,  Bx  =  c. 

The  exclusion  of  equalities  is  merely  to  facilitate  our  presentation. 

In  Section  2  we  will  study  some  properties  of  the  least-squares  solutions 
In  Section  3  we  will  present  the  method  and  discuss  its  relationship  with 
other  approaches.  Finite  convergence  theorems  will  be  established  for  the 
method  in  Section  4.  Numerical  results  will  be  given  in  Section  5. 


2.  THE  PROBLEM 


Let  A  be  an  m  x  n  matrix  and  b  be  an  m-vector,  we  consider  the 
system  of  linear  inequalities  Ax  <  b.  For  any  n-vector  x  there  is  a 
corresponding  m-vector  Z(x)  defined  by 

Z  (x)  :=  (Ax  -  b)  . 

The  vector  Z(x)  may  be  called  the  residual  of  the  vector  x  and  is  a 

measurement  of  violation  of  the  inequalities  Ax  <  b  at  x.  A  least-squares 

solution  x*  of  Ax  <  b  is  any  vector  that  minimizes  the  quantity  lz(x)l2# 

or  equivalently  the  function  f (x)  d  lined  in  (1.1).  We  note  that  the  function 

T 

f  is  differentiable  and  convex  and  its  gradient  Vf (x)  is  A  (Ax  -  b)  +  . 
Therefore  we  have  the  following  necessary  and  sufficient  condition  for  a 
least-squares  solution. 

THEOREM  2 . 1  A  n-vector  x*  is  a  least-squares  solution  of  the  system 
Ax  <  b  if  and  only  if 

AT(Ax  -  b)  =  0.  (2.1) 

Condition  (2.1)  may  be  called  the  normal  equation  of  the  system 
Ax  <  b  because  it  is  analogous  to  the  normal  equation  in  the  equality  case. 

The  relationship  between  a  least-squares  solution  and  the  quadratic 
program  (1.2)  is  fundamental  and  useful.  We  state  it  in  the  following 
theorem.  The  proof  is  trivial  and  hence  omitted. 

THEOREM  2.2  A  n-vector  x*  is  a  least-squares  solution  of  Ax  <  b  and 
and  z*  =  Z  (x*)  if  and  only  if  (x*,z*)  is  a  solution  of  the  quadratic 
program  (1.2). 

We  next  show  the  existance  of  a  least-squares  solution. 

THEOREM  2.3  A  least-squares  solution  of  Ax  <  b  exists. 

PROOF:  Because  of  Theorem  2. 2  we  only  need  to  show  that  a  solution  of  the 
quadratic  program  (1.2)  exists.  We  observe  that  the  feasible  region 
X  =  { (x,z) | Ax  -  z  <  b}  is  a  nonempty  polyhedral  set  and  the  objective  function 
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z  is  quadratic  convex  and  bounded  below  on  X.  Therefore,  it  follows  from 
Corollary  27.3.1  in  [6]  that  a  solution  exists.  □ 

It  is  shown  by  Mangasarian  [5]  that  McCormick's  second  order  sufficient 
optimality  condition  is  also  necessary  for  a  solution  of  a  quadratic  program 
to  be  locally  unique.  If  we  apply  this  result  to  (1.2)  then  we  have  the  follow¬ 
ing  theorem  on  the  uniqueness  of  least-squares  solution  to  the  system  Ax  <  b. 

We  first  note  that,  for  a  given  index  set  I  c  {1, ,m),  we  use  A  to  denote 

the  submatrix  of  A  that  consists  of  rows  in  I.  The  subvector  z  is  defined 

similarly. 


THEOREM  2 . 4  Let  x*  be  a  least-squares  solution  of  Ax  <  b  and  let 
T 

I  =  {i|a^  x*  =  b^}.  Then  x*  is  the  unique  least-squares  solution  to  the 

system  Ax  <  b  if  and  only  if  there  is  no  nonzero  x  satisfying  A^x  <  0. 

PROOF:  Recall  that  McCormick's  second  order  sufficient  optimality  condition 

T 

for  Problem  (1.2)  is  that  z  z  >  0  for  any  nonzero  pair  (x,z)  satisfying 

Ajx  “  zj  5  0 

z*Tz  =  0, 

where  z*  =  (Ax*  -  b)  +  .  Clearly,  this  condition  is  equivalent  to  the  system 

Ajx  <  0  having  no  nonzero  solution.  Therefore,  our  result  follows  directly 

from  Theorem  2.1  of  [51.  □ 

We  note  here  that  the  uniqueness  condition  in  the  above  theorem  is  also 

equivalent  to  the  following:  the  matrix  A^  has  full  column  rank  and  there 

T 

exists  a  positive  vector  u  satisfying  A  u  =  0.  These  conditions  are  too 
restrictive  and  least-squares  solutions  of  most  problems  in  practice  are  usually 
not  unique.  However,  for  a  given  system  Ax  <  b,  the  optimal  residual  vector 
z*  is  always  unique.  Vie  give  this  result  below. 

THEOREM  2.5  There  exists  a  unique  m-vector  z*  such  that  x*  is  a  least- 

squares  solution  of  Ax  <  b  if  and  only  if  (Ax*  -  b)  =*  z*.  Furthermore, 

T 

z*  >  o  and  A  z*  =»  0. 
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Hence,  we  have  the  following  corollary. 

CQRQLLAKy  2.6  There  exists  an  m-vector  z*  >  0  such  that  the  system  (2.2)  is 
consistent  and  x*  is  a  least-squares  solution  of  Ax  <  b  if  and  only  if  x* 
satisfies  (2.2). 

Using  a  similar  argument  as  in  the  proof  of  Theorem  2.4,  we  can  show  that 
the  residual  vector  of  the  equality  linear  least-squares  problem  is  unique. 

Since  this  result  will  be  used  later,  we  state  it  below  as  a  corollary  of  Theorem  2.4. 
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COROLLARY  2 . 7  Let  B  be  an  m  x  n  matrix  and  c  be  an  m-vector.  Uiere  exists 

a  unique  m-vector  r*  such  that  a  vector  x*  is  a  least-squares  solution  of 

T 

the  linear  equation  Bx  =  c  if  and  only  if  Bx*  =  c  +  r*.  Furthermore,  B  r*  = 
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3.  THE  METHOD 


The  proposed  method  is  an  iterative  process.  Having  an  estimate  x  of 

a  solution,  we  produce  a  new  estimate  x  by  first  finding  the  index  set  1 

i  T 

of  active  and  violated  inequalities  at  x.  That  is  I  =  { i  | a^  x  >  b.^}  where 
T 

a^  is  the  i-th  row  of  A  and  b^  is  the  i-th  componant  of  b.  We  then 


generate  a  search  direction  d  by 

+ 


=  -AI  (AjX  -  b  )  , 


(3.1) 


where  A^+  is  the  pseudo-inverse  of  A^.  Recall  that  A^+  is  the  unique 
matrix  satisfying  the  following  Moore-Penrose  conditions: 


(a)  ArAr  AI  =  Ax 


(C)  AI+AIAI+=AI+ 


(b)  <ViV  =  AIAI+ 

(d)  (A^A^7  =  A^Al 


(3.2) 


Once  the  search  direction  d  is  found,  we  determine  a  stepsize  X  by 

minimizing  the  function  0 (X )  ;=  f (x  +  Ad) .  Here,  K  is  chosen  to  be  the 

smallest  minimizer  of  0(A).  Then  the  new  estimate  x  is  set  to  be  x  =  x  +  Ad. 

T 

We  repeat  this  process  until  a  point  x*  satisfying  A  (Ax*  -  b)+  =  0 
is  found. 

We  note  here  that  to  compute  d  it  is  not  necessary  to  construct  the 

pseudo-inverse  A^.+  explicitly.  The  direction  can  be  generated  by  applying  a 

singular  value  decomposition  to  A  .  We  also  note  that  the  function  0(A) 

m  2 

is  of  the  form  0(A)  =  /  (a. A  -  7.)  .  Therefore,  if  a.  <  0  for  all  i  then 

. L ,  i  i  +  i  = 

i=l 

0(A)  becomes  a  constant  when  A  is  larger  than  a  certain  number;  and  if  >  o 
for  some  i  then  0(A)  ■*■<*>  when  A  -*■  00 .  Hence,  the  stepsize  A  can  not  be 
infinite.  Because  6  is  convex  and  piesewise  quadratic,  the  stepsize  A  can 
be  very  accurately  and  effeciently  computed. 

Recall  that  the  vector  d  defined  in  (3.1)  is  the  minimum-norm  least- 
squares  solution  of  the  linear  equation  A^d  =  b^  -  A^x.  To  gain  more  insight 
into  the  method,  we  give  the  following  theorem. 
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THEOREM  3 . 2  Let  x  =  x  +  d,  then  x  is  the  closest  least-squares  solution 
of  the  system  A^y  =  to  the  point  x  in  2-norm. 

PROOF.  We  first  show  that  x  is  a  least-squares  solution  of  AJy  =  b  .  It 

follows  from  (3.2)  that 

T  +  T  +  T  +TT 

VA  ■  VVi>  ■  'ViV  ■  *1  • 


This  implies  that 

A^Aj.x  =  A^A^x  +  d) 

=  A^x  ~  aXAI  {V  "  bI5 

T  T 

=  A  AX  -  AjfAjX  -  bI> 

T 

=  A1bI. 

Hence,  x  is  a  least-squares  solution  of  A^y  =  b^. 

Let  y  be  any  least-squares  solution  of  A^.y  =  b^  and  let  d  =  y  -  x. 

We  want  to  show  that  ||  d  ||  ^  <  !!  d  1 1  2 -  Because  d  is  the  minimum  norm  least- 

squares  solution  of  A^y  =  b  -  A^x,  we  only  need  to  show  that  d  is  a  least- 

squares  solution  of  Axy  =  b  -  AjX.  This  follows  from 

T  T  -  T 

Vid  =  \V  -  Vix 

T  T 

=  Aibi  "  W 

=  Aj (b j  -  AjX)  . 

The  above  theorem  shows  that  x  is  the  projection  of  x  onto  the  set  of 
all  least-squares  solution  of  AIy  =  b^ .  From  this  observation  the  vector  d 
is  a  very  natural  choice  for  a  search  direction. 

The  method  is  also  related  to  Newton's  method  for  minimizing  the  function  f. 


□ 


If  the  set  (i|  x  =  M  is  empty  then  the  function  f  is  twice  differentiable 

2  T 

at  x  and  its  Hessian  is  V  f(x)  =  AjAj.  If  we  further  assume  that  A  has 

2 

full  column  rank,  then  V  f(x)  is  nonsingular  and  the  direction  generated  by 
Newton's  method  is  given  by 
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d  =  -(AjAI)"1Vf (x)  =  -(AjAi)_1AT(Ax  -  b)+ 

T  -IT 

=  - (AjAj)  aJ^x  -  bj). 

+  T  -1  T 

Because,  in  this  case  we  have  AJ  =  (A^^)  A  ,  our  method  and  Newton's  are 
identical. 
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4.  FINITE  CONVERGENCE 


In  this  section  we  will  show  that  for  any  given  system  of  linear  inequalities 
Ax  <  b,  our  method  will  produce  a  least-squares  solution  in  a  finite  number  of 
iterations. 

We  first  show  that  the  direction  d  generated  in  (3.1)  is  a  descent  direction 


for  the  function  f  at  x. 

LEMMA  4.1  Vf  (x)  = 

PROOF  It  follows  from  Theorem  3.1  that  the  vector  £  =  x  +  d  satisfies  the 
T  T 

normal  equation  AjA^  =  Ajbj.  Thus 

Vf(x)  =  AT(Ax  -  b)+  =  Aj(AjX  -  bj) 

T  -  T 

=  AjAj  (x  -  d)  -  Ajbj. 


T  —  T  T 

=  -AjAjd  +  AlAl*  -  A;[bI 


-AIv- 


From  Lemma  4.1  we  immediately  have  the  following  theorem. 

THEOREM  4.2  Vf(x)Td  =  -  ||A,d||2  . 

— - — — — — —  "  I  "2 

We  then  show  that  for  any  sequence  {x  }  generated  by  the  method  we  have 
lim  Vf (xk)  =  0. 

LEMMA  4.3  For  any  u,v  in  Rn 

II  Vf(u)  -  Vf  (v)  ||  2  *  II  a||2  II  u  -  v||2  . 

PROOF  ||  Vf  (u)  -  Vf  (v)  ||  2  =  |1  AT(Au  -  b)+  -  AT(Av  -b)  +  || 

<  I!  AT||  2||  (Au  -  b)+  -  (Au  -  b)  +  ||  2 

<  ||  AT||  2||  Au  -  Av j|  2 

<  II  a||2  II  u  -  v||2  □ 
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THEOREM  4.4  Let  {x  }  be  generated  by  the  method  from  any  starting  point  x 


-  k  k 

Then  either  for  some  k  <  “,  Vf{x  )  =  0  or  lim  Vf(x  )  =  0. 

k-**> 

PROOF;  We  note  that  if  d  =  0  then  by  Lemma  4.1  we  have  Vf(x)  =  0  and  x  i 
a  solution.  Hence,  we  assume  that  x  is  not  a  solution  and  d  /  0.  This  also 
implies  that  I={i|aix>8i}/  0. 


Let 


c,  = 


C2 

A 


max  ||  A* || 
1/0 


-Vf (x)Td 


Then  we  have  that 

f(x  +  Ad)  -  f(x)  =  A/0Vf(x  +  tAd)Td  dt 

=  A [7f (x)Td  +  /Q((7f(x  +  tAd)  -  7f(x))Td)dt] 
<  A[Vf(x)Td  +  C;lA||  d||  ^  /J  t  dt] 

=  ^f(x)Td 

=  -(  f(x)Td)2 

Zcjldl^  . 


We  also  note  that  it  follows  from  (3.2.c)  that 

l|d||2=  II  <Axx  “  bT)|l2  =  ||  »JV||2 

<  ll^lljll  vi|a 


T- 

On  the  other  hand,  it  follows  from  Theorem  4.2  that  Vf  (x)  d 
Therefore,  we  have  that 


II  Mil! 


f  (x)  -  f  (x  +  Ad)  > 


1 


Because  of  the  choice  of  the  stepsize  X  and  x  =  x  +  Xd,  we  have  that 

f  (x)  -  f(x)  >  f  (x)  -  f  (x  +  Xd)  >  ——  ||  A  5 1 1  ?  . 

6  X  c 

2C1C2 

Since  the  sequence  {f(x  )}  is  monotone  decreasing  and  bounded  below,  we  have 


>  l  -  f (xk+1))  >  — ~  l  ||  A 


2c,c  k=0  k 
1  2 


5kll2 . 


This  implies  that 


lim  A  dk  =  0 
k-«°  Ik 

Then  it  follows  from  Vf (xk)  =  -A^  A  dk  that 

Ik  k 

lim  Vf(xk)  =  0  □ 

k-x° 

By  Theorem  2.4  there  exists  a  unique  vector  z*  such  that  x  is  a  least- 

squares  solution  of  Ax  <  b  is  and  only  if  (Ax  -  b)  -  z*.  If  the  sequence 

Jc  Jc 

{x  }  is  generated  by  the  method  and  z  =  (Ax  -  b)+,  we  want  to  show  that 

lim  z  =  z*.  To  do  this,  we  need  the  following  lemma.  The  proof  of  the  lemma 
k-+°° 

can  be  found  in  [2] . 

)c 

LEMMA  4.5  Let  B  and  E  be  any  m  x  n  and  £  x  n  matrices  and  let  (g  } 
and  {hk}  be  sequences  of  vectors  in  Rm  and  in  R^  respectively.  If 
g  -*•  g  and  h  -*■  h  and  for  each  k  the  system  Bx  =  g  ,  Ex  <  h  is 

consistent  then  the  system  Bx  =  g,  Ex  <  h  is  also  consistent. 

k  k  k 

THEOREM  4.6  Let  {x  }  be  generated  by  the  method  and  z  =  (Ax  -  b)+,  then 

lim  z  =  z*  , 

k-*» 


where  z*  is  the  optimal  residual  vector  defined  in  Theorem  2.4. 

k  ^  k  T  k  Jc 

PROOF:  We  have  that  f(x  )  =  —  (z  )  z  and  {f(x  )}  is  a  monotone  decreasing 
sequence  and  bounded  below.  Hence,  there  exists  a  p  >  0  such  that  for  each  k 

zk  e  u  *  (z|  ||  z||2  <  p} 


L 


T  jC  X  /v 

Let  {z  J}  be  any  convergent  subsequence  of  {z  }  and  let  z  -*■  z  . 

We  want  to  show  z  =  z*.  Let  P  =  {  i|  z.  >0}  and  L  =  (i|".  =  o}.  Because 

k.  k . 

z  =  (Ax  •*  -  b)+,  we  have  that  for  all  sufficiently  large  jf  the  system 


k. 


Ax  =  b  +  z 
P  P  P 


ALX  "  bL  +  ZL 


k. 

3 


*j 


has  x  as  a  solution  and  hence  is  consistent. 
4.5  that  the  system 


Then  it  follows  from  Lemma 


'  V  '  bP  4  zp 

is  also  consistent  and  has  a  solution,  x  say.  We  now  show  that  x  is  a 

least-squares  solution  of  Ax  <  b.  Notice  that  z  =  (Ax  -  b)+.  It  follows 

T  k  Jc 

from  Theorem  4.4  that  A  z  =  Vf  (x  )  0,  which,  in  conjunction  with 

k . 

z  -*■  z,  implies  that 

VUx)  -  AT(Ai  -  b)+  -  ATz  =  0. 

Hence,  x  is  a  least-squares  solution.  By  Theorem  2.4  the  optimal  residual 

vector  z*  is  unique,  hence  z  =  z*. 

k 

We  now  show  that  z  -+  z*.  If  it  is  not  the  case,  then  there  exists  an 

k .  k 

e  >  0  and  a  subsequence  {z  •3}  such  that  |)  z  ^  -  z*||  >  e  for  each  j. 

Because  {  z  J}  is  in  the  compact  set  u,  it  has  a  convergent  subsequence. 

By  the  first  part  of  the  proof  this  subsequence  converges  to  z*,  which  con¬ 
i'.  k 

tradicts  ||  z  J  -  z*||  >  e  for  each  j.  Hence,  we  have  z  -*■  z* .  d 

I  T 

In  the  sequel  we  use  I(x)  to  denote  the  index  set  { i  |  a^  x  >  1=0  at  x. 

]( 

We  will  show  that  if  the  sequence  (x  }  generated  by  the  method  does  not 

terminate  after  a  finite  number  of  steps,  then  for  all  sufficiently  large  k 
k  C  k+1 

we  have  that  I(x  )  #  I(x  ).  This  is  a  contradiction  because  we  have  only 
finite  index  sets.  To  establish  these  results  we  need  some  lemmas. 
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LEMMA  4.7  There  exists  an  e  >  0  such  that  if  I  is  an  index  set  and  I  =  I (x) 


for  some  x  satisfying  ||(Ax  -  b)  +  -  z*||  <  e  then  the  system 

Ajy  -  +  z*  is  consistent. 

it 

PROOF.  If  it  is  not  true,  then  there  exists  a  sequence  {x  }  satisfying 

(Ax  -  b)  ->  z*  and  for  each  k,  the  system  A  y  =  b  +  z*  is  inconsistent, 

+  k  k  k  Xk 

where  1^  =  I(x  ).  Because  there  are  only  finite  index  sets,  we  may  assume 

that  1^  is  fixed  and  for  some  I  ,  1^  =  I  for  each  k. 

k  k  k  k 

Let  z  =  (Ax  -  b)+,  then  the  system  =  b  +  z  has  x  as  a 

solution  and  hence  is  consistent.  It  follows  from  Lemma  4.5  and  z  z* 

that  Ajy  =  b^  +  z*  is  consistent,  which  is  a  contradiction.  Hence  the 

lemma  is  proved.  □ 

LEMMA  4.8  There  exists  and  e  >  0  such  that  if  x  is  a  point  satisfying 

||  (Ax  -  b)+  -  z*||2  <  e,  d  is  the  direction  generated  in  (3.1)  and 

x  =«  x  +  d  then  I(x)  o  I(x). 

PROOF.  Choose  a  positive  number  e  sufficiently  small  so  that  Lemma  4.7 


holds  and  | (Ax  -  b) 


<  e  implies  that  I(x)  =>  P  :■  f i  | z*  >  0}. 


Let  x  be  any  point  satisfying  ||(Ax  -  b)+  -  z*||2  <  e  and  let  I  -  I(x). 

Then  by  Lemna  4.7  we  have  that  the  system  A^y  =  b^  +  z*  has  a  solution, 

T  T 

v  say.  It  follows  from  I  o  P  and  A  z*  =  A^z*  =  0  that 

y  z  Ajbj  +  AjZ*j  =  Aibi"  Hence»  y  is  a  least-squares  solution 
of  Ajy  =  bj  and  z*  is  its  residual  vector.  Notice  that  x  =  x  +  d 
is  also  a  least-squares  solution  of  A^y  =  b^.  It  follows  from  Corollary 
2.6  that  the  residual  vector  is  unique.  Hence,  z*  is  also  the  residual 
vector  of  x  and  we  have  A^x  ■  b^  +  z*.  Because  z*  >  0,  we  have 
I(x)  o  I  =  I(x).  This  completes  our  proof.  O 

LEMMA  4.9  Let  x  not  be  a  least-squares  solution  of  Ay  <  b  and  let 
*  x  +  d  and  x  *  x  +  Xd.  Then 


x 


(a)  I  (x)  =  I  (x)  =»  x  is  a  least-squares  solution  of  Ay  <  b 

(b)  I(x)  ^  I(x)  =*  I(x)  ^  I(x) . 

PROOF.  (a)  If  I(x)  =  I(x)  =  I,  then 

AT(Ax  -  b)+  =  A^tAjX  -  bj)  =  0. 

Hence,  x  is  a  least-squares  solution  of  Ay  <  b.  Because  f(x)  <  f (x) , 
the  point  x  must  also  be  a  least-squares  solution  of  Ay  <  b. 

(b)  We  first  show  that  the  stepsize  \  e  (0,1].  It  follows  from  Theorem 
4.2  that  0 »  (0)  <  0.  Therefore,  because  e  is  convex,  we  only  need  to  show 
0» (1)  >  0. 

Let  I  (x)  =  I  u  J,  where  J  n  I  =  j8  and  J  ?  f).  Then,  we  have  that 

0 ' (1)  =  dTVf (x)  -  dTAT(A£  -  b) 

=  dTA^(Aii  -  +  ^AjtAjX  -  bj). 

=  d'V'CA  x  -  b  ). 

J  J  J 

Since  Ax  >  b  and  Ax  <  b  ,  we  have 

55  J  J  J 

A,d  =  A,x  -  Ax  >  b  -  b  =0. 

J  J  J  J  J 

Therefore,  it  follows  from  A  x  -  b  >  0  and  A  d>0  that  9*  (1)  >  0. 

J  J  -  J  * 

Hence,  we  have  \  e  (0,1]. 

We  next  show  that  I(x)  ^  I (x)  implies  I(x)  3  I(x).  we  have  that 

AjX  =  (1  -  \)Aj.x  +  AAjX 

>  (1  -  Mb  +  lb  =  b_ 

=  III. 

Hence,  I (x)  3  I(x). 

We  then  show  that  I (x)  ?  I(x).  Suppose  I (x)  =  I (x)  =  I.  Then,  we  have 
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least-squares  solution  after  a  finite  steps,  then  by  Theorem  4.6  we  have  that 


lim  z  =  z*.  Let  e  >  0  be  defined  as  in  Lemma  4.8  and  let  k  be  chosen  so 
k-*» 

that  for  any  k  >  k,  we  have  ||  z  -  z*||  <  e.  Then  it  follows  from  Lemma  4.8 


k  k+1  - 

and  Lemma  4.9  that  I  (x  )  <£  X(x  )  for  all  k  >  k.  This  is  certainly  impossible. 


hence  {x  }  must  terminate  in  a  finite  number  of  iterations. 
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5.  REMARKS  AND  NUMERICAL  RESULTS 

The  linear  least-squares  problem  of  equalities  has  been  widely  studied  and 

efficient  computational  methods  exist  [see,  for  example,  1,3  and  7].  However, 

with  very  wide  range  of  applications,  the  linear  least-squares  problem  of 

inequalities  has  not  been  throughly  investigated  as  it  should.  This  paper  is 

an  attempt  to  provide  for  this  problem  a  useful  computational  method. 

A  very  interesting  application  of  the  linear  least-squares  problem  is  to 

solve  linear  programming  problems.  Consider  the  following  linear  program 

T 

min  c  y 

y 

s .  t .  By  >  d 
y  >  0  . 

Its  dual  program  is 

.T 

max  d  u 
u 

T 

s.t.  B  u  <  c 

u  <  0. 

By  a  duality  theorem,  the  linear  program  and  its  dual  can  be  solved  by  finding 
a  pair  (y,u)  that  satisfies  the  following  inequalities: 


where 


Ax  <  b. 


(5.1) 


Then  our  method  can  apply  to  the  system  (5.1).  When  the  linear  program  has  no 
solution,  a  least-squares  solution  to  (5.1)  can  be  found.  It  is  also  interesting 


fl 


I 
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to  note  that,  by  doing  so,  we  can  transform  a  linear  programming  problem  into 
unconstrained  minimalization  of  the  convex  and  differentiable  function  f  defined 


in  (1.1).  A  similar  function  was  studied  by  Mangasarian  [4].  His  function  is 
designed  to  find  a  least  norm  solution  for  a  linear  program.  In  his  case  the 
least-squares  measurement  is  taken  in  the  domain  space  rather  than  the  range 
space. 

The  method  has  been  tested  for  many  randomly  generated  problems.  The 
search  direction  d  is  computed  by  using  a  singular  value  decomposition  sub¬ 
routine  in  LINPACK.  The  method  is  very  satisfactory  in  accuracy  and  number  of 
iterations.  For  each  test  problem  a  point  with  ||  AT(Ax  -  b)+|(2  1  ^  is 

found.  The  number  of  iterations  is  usually  much  less  than  the  size  of  the 
problem. 

In  the  following  table,  m  is  the  number  of  inequalities  and  n  is  the 
number  of  variables.  The  numbers  in  the  table  indicate  the  numbers  of  iterations. 
The  computation  is  done  in  cyber  175  System  at  the  University  of  Illinois  at  Urbana. 


Table 


\  n 
m  V 

10 

10 

20 

20 

30 

— 

30 

40 

40 

50 

50 

10 

5 

2 

2 

2 

3 

2 

2 

2 

2 

3 

20 

4 

9 

3 

3 

4 

3 

3 

4 

3 

2 

30 

4 

4 

8 

11 

2 

4 

5 

6 

4 

3 

40 

5 

5 

10 

10 

9 

8 

4 

6 

3 

5 
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